MEMORANDUM January 16, 2015 
TO: Board Members 


FROM: Terry B. Grier, Ed.D. 
Superintendent of Schools 


SUBJECT: |THE EFFECT OF THE BAYLOR COLLEGE OF MEDICINE SUMMER SCIENCE 
INSTITUTE ON HISD ELEMENTARY AND MIDDLE SCHOOL STUDENTS’ 
SCIENCE PERFORMANCE, 2013-2014 


CONTACT: Carla Stevens, (713) 556-6700 


A total of 345 HISD teachers participated in the Baylor College of Medicine Summer Science 
Institute (BCMSSI) which offered professional development in current and effective teaching 
strategies, in-depth science content lessons, assessment, and relevant science concepts aligned 
to Texas Essential Knowledge and Skills (TEKS) and State of Texas Assessments of Academic 
Readiness (STAAR) recommended learning objectives. The purpose of this report was to 
measure the effect of the BCMSSI on the science performance of students whose teachers 
participated in the program during summer 2013. 


Paired t-test results showed statistically significant increases in the BCMSSI students’ Stanford 
10 science performance at the third, fourth, and eighth grades. The results also showed moderate 
to substantively positive BCMSSI program effects, .20, .31 and .32, respectively, on these grade 
levels. 


Linear regression analysis indicated that students’ previous year, (2013) Stanford 10 science 
scores were the strongest predictors of science performance. At-risk status predicted science 
performance in all but the third grade and special education in all but the seventh grade, reflecting 
lower mean science scores. Summer hours positively predicted science scores in the fourth, fifth, 
and eighth grades and follow-up professional development hours positively predicted the science 
performance of third and fourth-grade BCMSSI students. 


Implications include additional mentoring and science support throughout the school year, and 
close monitoring of the instructional practices BCMSSI teachers use in the classroom. Future 
BCMSSI may also focus on specific strategies designed to meet the specialized needs of HISD 
at-risk, economically disadvantaged, and special education students. 


Administrative Response: The Baylor College of Medicine Summer Science Institute will 
continue as an option for professional development for elementary and middle school teachers. 
At the same time, the program does warrant further investigation to determine (a) the extent to 
which teachers’ instructional practices can be measured to show the use of the programs’ 
activities and strategies and how that might impact results; (6) more effective ways of designing 
activities and strategies to address specialized groups to improve the negative beta Standard 10 
scores for special education, economic-disadvantaged, and at-risk students; and (c) how the 
inclusion of future longitudinal tracking of science interest and knowledge would affect results. 


Should you have any questions or require any further information, please contact me or Carla 
Stevens in the Department of Research and Accountability, at 713-556-6700. 
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The effect of the Baylor College of Medicine Summer Science Institute on HISD 
elementary and middle school students’ science performance, 2013-2014 


By Ted D. Serrant, PhD. 


The purpose of this report was to measure the effect of the Baylor College of Medicine Summer Science Institute 
(BCMSSI) on science performance of students whose teachers participated in the program during summer 2013. 
Paired t-test results showed statistically significant increases in the BCM students’ Stanford 10 science 
performance at the third, fourth, and eighth grades. Results also showed moderate to substantively positive 
BCMSSCI program effect at these grade levels. Linear regression analysis indicated that students’ previous year 
Stanford 10 science scores were the strongest predictor of science performance. Implications include additional 
mentoring and science support throughout the school year, and close monitoring of the instructional practices that 


BCMSSI teachers use in the classroom. 
Background 


The BCM Summer Science Institute (BCMSSI) is a 
professional development program that offers current 
and effective teaching strategies, in-depth science 
content lessons, assessment, and relevant science 
concepts that are aligned with the Texas Essential 
Knowledge and Skills (TEKS) and the State of Texas 
Assessment of Academic Readiness (STAAR) 
recommended learning objectives (BCM, 2013). This 
program targeted the Houston Independent School 
District (HISD) prekindergarten to eighth-grade 
teachers who taught science in self-contained classes, 
science labs, and regular education classes. The 
Institute included presentations by scientists and the use 
of inquiry-based, grade-appropriate science lessons. 
Hands-on lessons and activities with online lessons 
were available for workshop participants and other 
teachers. 

New BCMSSI participants attended a two-week 
workshop. Returning elementary- and middle-school 
teachers attended a one-week workshop. In addition, 
teachers were encouraged to attend five follow-up 
workshop sessions throughout the academic year. The 
program culminated with a science festival in which 
participants displayed skills and content learned. 


Literature Review 


Participation in summer science institutes have 
increased the number of teachers providing 
opportunities for their students to conduct full 
scientific inquiry using deep science content and 
process knowledge with numerous opportunities for 
practice (Jeanpiere, Oberhauser, & Freeman, 2005). 
Where teachers examined student thinking and their 
implications for instruction, student and teacher 
science content knowledge and test scores increased 
(Heller, Dahler, Wong, Shinohara, & Miratrix, 2012). 

Research has shown that teachers who demonstrated 
improvements in science content knowledge and their 
abilities to analyze science teaching also demonstrated 
higher gains in student science content knowledge 
(Roth, et al., 2011). The improvement in student science 
content knowledge was also associated with teachers’ 
pedagogical content knowledge about student thinking, 
and teaching practices aimed at improving the 
coherence of science content storylines (Roth, Garnier, 
Chen, Lemmens, Schwille, & Wickler, 2011). 

Third- and fourth-grade culturally and linguistically 
diverse students whose teachers had a year-long science 
professional development program demonstrated 
significant growth in the inquiry abilities of all students. 
Low-achieving, low-SES, and LEP students made 
impressive gains (Cueras, Lee, Hart, & Deaktor, 2005). 
A three-year professional development program to 


improve teaching practice significantly raised third- to 
fifth-grade science scores. It consistently increased the 
achievement gaps among the third- and fifth-grade 
demographic groups but held steady for the fourth grade 
(Luft, Wong, & Ortega, 2009). 

A study designed to measure the influence of inquiry 
science on teacher practices among fourth- and fifth- 
grade teachers in the Los Angeles United School 
District found that there was increased incidence of 
inquiry-based science teaching. The impacts, however, 
were limited to selected features of the inquiry process 
to which the teachers were most frequently exposed 
during the professional development (Griggs, Kelly, & 
Geoffrey, 2013). 

Teachers exposed to long-term professional 
development programs demonstrated increased content 
knowledge. Teachers in the four-year Rice Elementary 
Model Science Lab (REMSL) increased their scientific 
content knowledge in all four years. In the last two 
years of the program, their gains in science content 
knowledge, use of inquiry based instruction, and 
leadership skills were significantly higher than that of 
the control group participants (Deaconu, Juskavceric, & 
Nichol, 2012). Students’ science process skills, and 
their content and concept knowledge was assessed after 
one-year of participation in an elementary STEM 
professional development program. Results showed 
gains in science process skills, science concepts and 
content knowledge among the general education 
students. Teacher participation in the program had 
statistically significant effects on student posttest 
science scores (Cotabish, Dailey, Ann, & Hughes, 
2013). 

The 2011 and 2012 evaluation of the BCMSSI’s 
impact on elementary and middle school science 
performance found statistically significant increases in 
the Stanford 10 Science scores of third- to fifth-grade 
students in the 2012 sample and fifth-grade in the 2013 
sample. Similar increases existed among the seventh 
and eighth-grade students whose teachers participated 
in the PD program. Linear regression modeling 
indicated that the follow-up or additional hours of 
professional development was the strongest predictor of 
students’ science performance in 2012 and 2013 
(Holmes, 2013). 

The purpose of this report is to measure the effect of 
the BCMSSI on the science and environmental science 
performance of elementary and middle school students 
whose teachers participated in the Institute training 
activities. Specifically, the study sought to answer two 
questions: 

1. How did students whose teachers participated in 


the 2013 BCM Summer Institute perform in 
science? 
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2. What was the effect of the 2013 BCM Science 
Summer Institute on students’ science 
performance? 


Methodology 


Data Analysis 

A repeated measures design of students’ pre- and 
post-intervention STAAR and Stanford 10 Science and 
environmental science test scores were used to measure 
performance based on teachers’ participation in the 
BCMSSI professional development program. Students 
2013 and 2014 test results were included in the study. 
In addition, their teachers had to have participated in the 
Summer Institute. Those who did not possess both 
scores and whose teachers did not participate in the 
Summer Institute were dropped from the sample. The 
2013 test scores were treated as the pretest, and the 
2014 test score were treated as the posttest scores. 
Using data from the Public Education Management 
Information System (PIEMS), State of Texas 
Assessment of Academic Readiness (STAAR), and 
Stanford 10 databases, students were matched with 
their teachers who participated in the BCMSSI and their 
science scores analyzed. The effect size of the program 
on the students’ science performance was determined 
by grade using Hedges’ g statistic. Hedges’ g is a 
standard deviation-based measure of a program effect. 
The U.S. Department of Education’s Institute of 
Education Sciences What Works Clearinghouse 
(WWC) and the Texas Education Agency (TEA) 
determined that 0.25 is a substantively-important effect 
size in education (Texas Education Agency, 2011). 


Linear Regression Analysis 

Pretest scores, summer PD hours, PD follow-up 
hours, and key demographic variables were analyzed to 
determine the factors that contributed to the variability 
in the students’ science scores and, therefore, predict 
their science performance. Demographic variables 
included economic status, gifted and _ talented 
identification status (G/T), at-risk status, special 
education status, and gender. 


Sample 

A total of 345 elementary and middle school teachers 
participated in the 2013 BCMSSI. Of these, 189 
(45.8%) were enrolled in the 60-hour Summer Institute 
and four follow-up sessions. About 68.8% (130 
teachers) completed all 60 hours of professional 
development. Additionally, 66 middle school teachers 
and 89 elementary school teachers enrolled in the 30- 
hour Summer Institute. Of these 56 (84.8%) and 71 
(79.7%), respectively, completed the 30-hour summer 
sessions. Table 1 displays the demographic 
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Table 1. Students Demographic Characteristics By Grade 


Grade Level 
First Second Third Fourth Fifth Sixth Seventh Eighth 

Total students 611 635 529 621 298 342 329 1135 
% % % % % % % % 
Gender: Male 52 52 52 49 51 51 52 50 
Female 48 48 48 51 49 49 48 50 
Race/ethnicity: 61 54 33 29 36 32 31 30 

Black 
Hispanic 33 39 65 67 56 64 55 62 
White 4 5 2 2 6 3 10 5 
Non-At-Risk 19 22 53 44 55 43 53 40 
At-Risk 80 78 47 56 45 57 47 60 
Non-Special Ed. 91 92 92 91 91 87 92 92 
Special Ed. 9 8 8 9 9 14 8 8 

Non-G/T 88 85 80 719 82 91 85 85 
G/T 12 15 20 21 19 9 15 15 
Non-LEP 92 89 62 55 69 77 87 84 
LEP 9 12 38 45 31 23 13 16 
Non-Econ. Disadv. 14 15 9 9 12 8 27 21 
Econ Disadv. 86 85 91 91 88 92 73 79 


composition by grade for students whose teachers 
participated in the Summer Institute. 


Variables of Interest 

The variables of interest include students’ STAAR 
science and Stanford 10 environment and science test 
scores. STAAR is administered to fifth- and eighth- 
grade HISD students. Stanford 10 environment is 
administered to first- and second-grade students and 
Stanford 10 science to third through eighth-grade 
students. Teacher total summer and follow-up sessions 
were included in the analysis. Where students had more 
than one teacher, the average number of teacher hours 
was used. Demographic variables from PEIMS 
included gender, LEP, special education, economic 
status, at-risk for dropout status, and ethnicity were also 
included in the analysis, and were disaggregated by 
grade and school levels (elementary and middle 
schools). 

The data met the normality, homoscedasticity and 
collinearity conditions using the Shaphiro-Wilk test, the 
normal Q-Q plot and the Detrended normal Q-Q plot on 
the IBM Statistical Package for Social Sciences (SPSS) 
software. 


How did the students whose teachers participated in 
the 2013 BCM Summer Institute perform in 
science? 


Stanford 10 and STAAR science test results were 
used to measure the impact of the BCMSSI on science 
achievement. The results are presented by elementary 
and middle school levels for students whose teachers 
participated in the Summer Institute. 


Table 2. Elementary Student 2014 Stanford 10 
Performance of BCM 2013 Summer Institute Teachers 


N Mean Std. 
NCE 
Environment 
1st 613 41.7 21.5 
one 635 44.9 16.1 
Science 
3rd 529 50.5 21.4 
4th 621 54.3 20.3 
5h 298 46.0 20.9 


Note: 40 - 60 NCEs is Average (Stanford 10 Manual) 


Table 2 shows that at the elementary level, first- 
through fifth-grade student performance fell within the 
average range on the Stanford 10 environment science 
and science subtests (between 40.0 and 60.0 NCEs). 

Among the third through fifth-grade, fourth-grade 
students attained the highest average on the Stanford 10 
science subtest (54.3 NCEs) at the elementary level. 
Second-grade students attained a higher mean NCE 
than the first-grade students on the environment 
Science subtest (41.7 NCEs vs. 44.9 NCEs). 


Table 3. Middle School Student 2014 Stanford 10 
Performance Of BCM 2013 Summer Institute Teachers 


N Mean Std. 
NCE 
Grade 
6m 325 47.3 22.1 
7 315 52.4 21.6 
gh 1088 55.0 19.5 


Note: 40 - 60 NCEs is Average (Stanford 10 Manual) 


Stanford 10 science subtest results for sixth through 
eighth-grade middle school students whose teachers 
participated in the 2013 BCM Summer Science Institute 
are presented in Table 3. The mean NCE at each grade 
level fell within the Stanford 10 average range (between 
40-60 NCEs). The 8"*-grade students included in the 
analysis had the highest performance (55.0 NCEs) 
compared to sixth- and seventh-grade students (47.3 
and 52.4 NCEs, respectively). 

The fifth- and eighth-grade STAAR science 
performance of students whose teachers participated in 
the 2013 Summer Science Institute were analyzed using 
the scale scores and Level II: Satisfactory, Phase-In 1 
performance standards. 


Table 4. 2014 STAAR Mean Scale Score and Percent 
Who Met Standards, Fifth and Eighth Grade 


Mean Std. Level IT: 
Scale Deviat. | Satisf. Phase — 
Score In 1 Standard 
*5th Grade 3676 460.8 65.5 
(n= 4062) (3720) (67.0) 
**gth Grade 3704 697.5 62.1 
(n = 2593) (3750) (64.0) 


*5'h grade — “Satisfactory” Scale Score: 3500-4306 
**8th orade — “Satisfactory” Scale Score: 3500-4327 
(Texas Education Agency, 2014) 

Comparative HISD data are given in parentheses 


Table 4 shows the mean STAAR performance of the 
fifth- and eighth-grade students and the proportion of 
students that fell within the Satisfactory performance 
range. The 2014 mean science scale score for the 
eighth-grade students was higher than the mean science 
scale score of the fifth-grade students. In addition, the 
proportion of the fifth graders who met the Level II: 
Satisfactory, Phase-In 1 standard on STAAR was 
higher than the performance of the eighth-grade 
students (65.5% vs. 62.1%). However, performances 
for both grades remain below the districts’. 


What was the effect of the 2013 BCM Summer 


Institute on students’ science performance? 


Paired t-tests were conducted to determine the effect 
of the BCM Summer Institute on the mean science 
NCEs of elementary (third to fifth grade) and middle 
school (sixth to eighth grade) students whose teachers 
were enrolled in the BCMSSI. Table 5 and Table 6 in 
Appendix A (p. 7) display the program effect sizes 
using Hedges’ g. 

As depicted in Table 5, the mean difference between 
the pre- (2013) and posttest (2014) mean science NCEs 
for third through fifth grades were statistically 
significant (p < .001). The program, however, had a 
small to moderate effect on the third-grade science 
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(0.20), but a substantively positive effect (0.31) on the 
fourth-grade science performance. The program had a 
substantively negative effect (-0.39) on the fifth-grade 
mean science NCEs. 

The mean NCE difference between the science pre 
and posttest scores for the sixth and eighth grades were 
statistically significant as shown in Table 6. The BCM 
Summer Institute had a substantively positive effect on 
the eighth-grade mean science NCE (0.32), but a 
substantively negative effect on the sixth- and seventh- 
grade mean science NCEs. 

Regression analyses indicated that the 2013 Stanford 
10 science NCE and G/T identification were positive 
predictors of science performance at the third- through 
eighth-grade levels (p <. 001 and .05) (see Table 7 and 
Table 8, Appendix A, p. 8). At-risk status predicted 
science performance in all but the third grade. 
Similarly, special education predicted science 
performance in all but the seventh grade. Students’ 
economic-disadvantaged status also predicted science 
performance at the fifth, seventh, and eighth grades. 
These three variables had negative beta scores 
indicating lower mean science performance for special 
education, economic-disadvantaged, and at-risk 
students compared to their counterparts. 

The report also showed that summer PD hours were 
positive predictors of the 2014 Stanford 10 science 
performance for BCMSSI students in the fourth, fifth, 


and eighth grades. Additional professional 
development hours were positive predictors of 
BCMSSI | students’ 2014 Stanford 10 science 


performance in the third and fourth grade (p <. 001 or 
.05). 

Overall, the third through fifth-grade regression 
model accounted for 58% to 65% of the variance in the 
2014 Stanford 10 mean science NCEs. The model also 
accounted for 52% to 56% of the variance in the 2014 
sixth to eighth-grade Stanford 10 mean science NCEs. 


Discussion 


This study showed that the 2014 Stanford 10 science 
performance of students whose teachers participated in 
the 2013 BCMSSI was well within the average 
performance range (40 — 60 NCEs) for first through 
eighth-grade students. The BCMSSI had a moderate to 
substantively positive effect on the 2014 third-, fourth- 
and eighth-grade Stanford 10 mean science NCEs for 
students whose teachers were enrolled in the program. 
More than 60% of the fifth- and eighth-grade students 
(66% and 62%, respectively) whose teachers 
participated in the program met state standard, Level II: 
Satisfactory (Phase-In 1) on the STAAR science test. 
District-wide, however, 67% and 64% of fifth and 
eighth-grade students, respectively, met state standards 
in science. 


The study showed that students’ performance on the 
2013 Stanford science test was the strongest predictor 
of performance on the 2014 test for third through fifth- 
grade students whose teachers participated in the 
BCMSSI. Students’ prior science knowledge, in this 
study, therefore, appeared to be critical for their 2014 
Stanford 10 science performance. G/T identification 
was also a strong predictor of third through fifth-grade 
BCMSSI students’ science performance. 

In several grades, at-risk, special education and 
economic-disadvantaged status were negative 
predictors of the science performance of students whose 
teachers participated in the BCMSSI. It is unclear the 
extent to which the Institute activities, lesson content 
and strategies were specifically designed to address the 
needs of these specialized groups. 

The study lacked a direct measure of teacher 
instructional practices. Consequently, teachers’ BCM 
Summer and Follow-up participation hours were used 
as proxies. The study was also limited by the lack of a 
matched comparison group, that is, students of teachers 
who were motivated to enroll in the BCMSSI, but did 
not enroll or enrolled and did not participate. A 
repeated-measures design was used to determine the 
program effect using paired t-test, which provides a 
robust analysis considering data limitations. 

Future analysis should include longitudinal tracking 
of science interest and knowledge, particularly for 
elementary student groups. Additional professional 
development follow-up may be required to continue to 
improve students’ science performance. 

Close monitoring of the instructional practices 
BCMSSI teachers used to support their students, to 
examine students’ thinking, and to inform and analyze 
science instruction against the science TEKS and 
learning objectives is recommended to better inform 
BCMSSI follow-up and future summer programs, and 
to measure teacher effect. Future BCMSSI may focus 
on specific strategies designed to meet the specialized 
learning needs of disadvantaged student are at-risk, 
economic-disadvantaged, and special education 
programs. 
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Table 5. Elementary Paired T-Test Analysis of 2013-2014 Student Cohort With Stanford 10 Science Data in 


2012-2013 
Effect Size 
Grade Std. Mean (Hedges’ 
Level Test Year Sample Mean Dev Diff. t p g) 
Pretest 2013 378 44.2 21,0 4.2 5.1 | .000* 0.20 
3 | Posttest 2014 378 48.4 20.0 
Pretest 2013 484 50.3 .0.00* 0.31 
4 _ | Posttest 2014 484 56.6 
Pretest 2013 292 55.5 .000* -0.39 
5 | Posttest 2014 292 46.4 | 24.2 


Note: *p < .001 


Table 6. Middle-School Paired T-Test Analysis of 2013-2014 Student Cohort With Stanford 10 Science Data 
in 2012-2013 


Effect Size 
Mean (Hedges’ 
Grade Level Test Year Sample Mean Std. Dev. | Diff. t p g) 
Pretest 2013 325 54.1 20.3 -6.9 | -7.7 | .000* -0.33 
6 | Posttest | 2014 325 47.2 22.1 
Pretest 2013 315 54.1 063 -0.07 
7 | Posttest | 2014 315 52.5 
Pretest 2013 1088 48.7 21.2 6.3 | 13.9 |.000* 0.32 
8 | Posttest | 2014 1088 55.0 19.5 


Note: *p < .001 
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Table 7. Linear Regression Modeling for Elementary School Students (3" to 5 Grades), 2014 


2014 Stanford 3" 2014 Stanford 4" 2013 Stanford 5" Grade 
Variable Grade Science NCEs Grade Science NCEs Science NCEs 
n= 378 | R?=58% | n=484 R? = 65% n= 292 R? = 59% 
B P B P B P 
Economic-Disadvantaged .003 925 014 .674 -.138 .002* 
2013 Stanford Science NCEs .615 .0O00** .576 .OOO** 426 .000** 
Summer PD .057 .126 .207 .000** -.156 .000** 
Additional PD .094 .018* .094 .000** .066 115 
At risk status .036 348 -.101 .003* -.179 .000** 
Special Ed -.138 .000* * -.075 .032* -.119 .005* 
G/T 119 .003* 135 .000** .203 .000** 


Note: *p < .05, ** p< .001 


Table 8. Linear Regression Modeling for Middle School Students (6" to 8" Grades), 2014 


2014 Stanford 6" 2014 Stanford 7" 2013 Stanford 8"! Grade 
Variable Grade Science NCEs Grade Science NCEs Science NCEs 
n=325 | R?=53% | n=315 | R?=52% | n=1088 R? = 56% 
Dp P P 
Economic-Disadvantaged -.044 261 -.113 .002* -.045 .034* 
2013 Stanford Science NCEs .503 .0O0** 564 .000** 523 .000** 
Summer PD -.026 484 .063 .068 -.050 .014* 
Additional PD -.025 511 .026 052 0.15 458 
At risk status -.157 .002* -.135 .002* -.151 .000** 
Special Ed -.172 .000** -.017 645 -.106 .000** 
G/T .097 .020* .167 .000** 133 .000** 


Note: *p < .05, ** p< .001 


